Real Time Augmentation of Omni-directional Images with Consistent Lighting

Panoramic images can be used as an image-based method for navigating in a real environment. Recent publications have shown that it is possible to augment photographs with correct lighting. We combine these two approaches to an interactive tool with real-time modifications of the real scene maintaining consistent lighting. The possible changes are: Insertion of virtual objects, new light sources and changes of materials. The only user input is a high-dynamic-range photograph of a light probe. A 3D model is reconstructed from the light probe image using object-based reconstruction methods. The differential rendering for all user manipulations is performed with the latest generation of graphics hardware. Finally, we remove the restriction to a fixed viewpoint by projecting the environment map to the geometry in the fragment program and we present a hole-filling algorithm for the parts of the geometry which were not visible in the light probe image